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Abstract 

Air  combat  scenarios  present  unique  problems  where  the  solution  is  not  obvious.  An 
advanced  networked  simulation  environment  can  be  used  to  train  high-level  cognitive  air 
combat  skills  such  as  problem  solving.  A  problem  in  the  development  of  principled  and  con¬ 
struct  oriented  assessment  is  related  to  determining  the  level  of  specificity  of  the  assessment. 
We  present  a  detailed  discussion  of  the  definition  of  critical  competencies  associated  with 
combat  mission  problem  solving  performance,  and  we  describe  our  approach  to  develop  and 
implement  an  embedded  performance  assessment  system  that  maps  outcomes  to  these  com¬ 
petencies.  Finally,  we  discuss  the  implications  for  our  approach  for  comprehensive  assessment 
and  discuss  our  goals  for  an  evaluation  of  the  competency-based  approach  to  assessment. 
©  2002  Elsevier  Science  Ltd.  All  rights  reserved. 


1.  Introduction 

Air-to-air  combat  typically  involves  a  flight  of  multiple  friendly  aircraft  engaging 
multiple  adversary  aircraft  at  beyond  visual  ranges  (BVR).  United  States  Air  Force 
(USAF)  air  combat  pilots  extensively  train  to  learn;  aircraft  systems,  emergency 
procedures,  tactics,  air  combat  jargon,  weapons  capabilities,  adversary  capabilit¬ 
ies,  and  rules  of  engagement  (ROE).  A  goal  of  this  training  is  to  have  these  knowl¬ 
edges  and  skills  become  automatized  in  the  classroom  and  simulation  environments 
so  that  wartime  implementation  of  these  knowledges/skills  is  second  nature. 


*  The  views  expressed  in  this  paper  are  those  of  the  authors  and  do  not  necessarily  reflect  an  official 
position  of  the  organization  with  which  they  are  affiliated. 
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However,  this  does  not  necessarily  fully  prepare  those  pilots  for  the  unique  prob¬ 
lems  air  combat  engagements  pose.  The  number  of  adversary  aircraft,  types  of  air¬ 
craft,  inter-aircraft  geometries  (i.e.,  spatial  relationship  among  relevant  aircraft), 
political  influences,  and  pilot  skill  differ  for  each  air  combat  engagement.  Therefore, 
every  engagement  presents  a  unique  problem  solving  challenge.  Working  within 
each  individual’s  competencies  and  together  as  a  cohesive  unit,  the  pilots  must 
problem-solve  these  engagements.  “Problem  solving  is  cognitive  processing  directed 
at  achieving  a  goal  when  no  solution  method  is  obvious  to  the  problem  solver” 
(O’Neil,  1999).  These  air  combat  knowledges  and  skills  in — stark  contrast  to  the 
classroom,  stand-alone  simulator,  or  other  more  mundane  training  environment — 
must  be  combined  with  other  skills  and  team  members  in  a  dynamic,  hostile,  time- 
critical  environment  to  solve  a  complex  problem  where  the  goal  is  to  protect  friendly 
assets  and  eliminate  hostile  forces,  but  the  solution  is  not  always  obvious. 

The  USAF  conducts  training  exercises  such  as  “Red  Flag”  using  flying  aircraft  to 
specifically  provide  air  combat  pilots  with  an  opportunity  to  practice  these  higher- 
level  cognitive  processing  skills.  However,  these  exercises  frequently  require  more 
than  a  half  dozen  real  planes  to  fly  in  a  restricted  area  with  a  host  of  safety  regula¬ 
tions  affecting  the  realism.  Since  such  opportunities  are  scarce  and  limiting  because 
safety  and  security  restrictions  prevent  pilots  from  performing  as  they  would  in 
actual  combat,  it  is  desirable  to  have  environments  where  higher  order  cognitive 
problem-solving  skills  can  be  trained  and  measured.  A  realistic,  networked  simula¬ 
tion  environment  could  afford  these  opportunities  to  train  and  measure  higher  order 
cognitive  air  combat  skills. 

Distributed  mission  training  (DMT)  is  a  revolutionary  training  environment 
composed  of  real  (live  aircraft),  virtual  (manned  aircraft  simulators),  and  con¬ 
structive  systems  (computer  generated  forces)  that  allow  aircrew  members  to  train 
both  individually  and  collectively  (Carroll,  1999).  The  DMT  environment  is  based 
on  a  wide  area  network  of  individual  cockpits  and  visual  systems  that  permits 
interactive  training  as  single  aircraft  and  multiple  aircraft.  DMT  provides  a  com¬ 
prehensive  environment  of  simulation  technology  in  which  information  is  dynami¬ 
cally  shared  and  used  among  a  group  of  individuals  engaged  in  real-time  training 
scenarios  (Carroll,  1999).  DMT  allows  multiple  players  at  multiple  sites  to  engage  in 
training  scenarios  ranging  from  individual  and  team  participation  up  to  full  theater- 
level  (i.e.  multi-team,  multi-force)  battles.  Additionally,  computer-generated,  or 
constructive,  forces  can  be  used  to  substantially  enhance  the  training  scenario.  By 
combining  simulation  systems  through  computer  networking,  multiple  operators 
from  different  military  platforms  (e.g.  command  and  control  centers,  aircraft,  ships, 
tanks)  can  all  train  together,  adding  to  the  fidelity.  Real-time  simulation  systems 
such  as  DMT  allow  nearly  unlimited  training  opportunities  for  small  or  large-scale 
forces  from  their  own  location  or  a  deployed  training  site. 

Virtual  environments  such  as  DMT  offer  researchers  the  opportunity  to  manip¬ 
ulate  key  characteristics  of  the  training  environment  to  promote  learning  and  mas¬ 
tery  of  complex  interdependent  tasks  (Crane,  Robbins,  &  Bennett,  2000), 
significantly  improving  the  capability  to  train  at  the  mission  and  team  level  (Crane, 
1999).  As  part  of  our  research  program,  we  are  attempting  to  address  a  number  of 
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challenges  and  needs  associated  with  both  the  development  and  evaluation  of  DMT 
as  a  high-fidelity  virtual  environment  for  training  and  performance  assessment. 
Three  specific  areas  are  of  particular  interest  in  our  training  research  program. 
First,  developing  and  validating  methods  to  identify  and  define  mission  and  train¬ 
ing  requirements  for  individuals  and  teams.  Second,  developing  methods  for  linking 
requirements  to  instructional  strategies  and  principles  that  can  be  embedded  in  rea¬ 
listic  training  scenarios  within  the  training  environment.  Third,  evaluating  the 
impact  of  instruction  on  addressing  the  requirements  through  in-training  and  field 
performance  assessment. 

A  major  problem  in  the  development  of  principled  and  construct  oriented  assess¬ 
ment  is  related  to  determining  the  level  of  specificity  of  the  assessment.  A  second 
problem  is  developing  outcome  measures  that  are  both  construct  oriented  and  that 
permit  near-real  time  assessment  for  diagnosis  and  remediation. 

With  respect  to  the  first  problem,  some  researchers  have  made  a  move  towards 
examination  of  competencies,  which  may  differ  for  individuals  and  teams  (Cannon- 
Bowers,  Tannenbaum,  Salas,  &  Volpe,  1995).  Team  competencies  can  be  divided 
into  team-specific  and  task-specific  competencies.  Team-specific  competencies  may 
only  apply  to  a  particular  team,  yet  encompass  all  tasks  the  team  performs.  Task- 
specific  competencies  may  only  apply  to  certain  tasks.  Applied  to  aircrew  individual 
or  team  training  performance,  these  competencies  become  Mission  Essential  Com¬ 
petencies  or  MECs  (Colegrove  &  Alliger,  2002).  Mission  Essential  Competencies  are 
higher-order  individual,  team,  and  inter-team  competencies  that  a  fully  prepared 
pilot,  crew  or  flight  requires  for  successful  mission  completion.  For  example,  for 
“Air  Superiority”  (this  mission  assures  that  friendly  forces  have  dominant  control  of 
the  skies),  the  Engagement  Phase  MEC  reads:  Employs  ordnance  against  valid  hostile 
targets  and/or  denies  enemy  weapons  in  accordance  with  stated  mission  objectives. 
Each  MEC  is  further  defined  by  identifying  a  purpose,  and  a  beginning  and  ending 
point.  The  MEC  framework  also  includes  some  (more  general)  supporting  compe¬ 
tencies,  and  lists  of  knowledge  and  skills.  In  addition,  a  list  of  experiences  through 
which  these  knowledge,  skills,  and  competencies  can  be  learned  has  also  been  gen¬ 
erated.  Mission  Essential  Competencies  are  not  abstract  knowledge  or  general  skills. 
They  are  demonstrated,  to  a  degree  of  competency,  in  the  context  of  an  actual  mis¬ 
sion  or  high-fidelity  simulated  mission  (Colegrove  &  Alliger,  2002).  Individual  and 
team  competencies  are  critical  for  successful  missions  in  a  real  combat  environment. 

Using  the  MECs  as  the  guide,  DMT  enables  researchers  to  perform  systematic 
scenario  development  and  performance  assessment  to  identify  deficient  compe¬ 
tencies  and  subsequently  train  our  warfighters  in  those  deficiencies  before  they  head 
off  for  battle.  Our  approach  is  based  on  recommendations  made  by  Cannon-Bowers 
and  Bell  (1997)  regarding  the  need  for  knowledge-rich,  high-fidelity  environments  to 
foster  complex  skill  acquisition. 

Different  levels  of  detail  have  emerged  during  our  Mission  Essential  Competency 
development.  High-level  information  summarizes  what  we  have  called  Mission 
Essential  competencies.  Table  1  shows  the  MECs  for  very  top  level  training  objec¬ 
tives  associated  with  activities  such  as  finding  the  target,  fixing  it  in  terms  of  its  exact 
location  and  direction,  tracking  it  to  ensure  you  know  where  its  going,  targeting  the 


776 


\\  .  Bennett  Jr.  et  al.j  Computers  in  Human  Behavior  18  (2002)  773—782 


Table  1 

Top  level  training  objectives  by  mission  essential  competency  linkage 


entity,  engaging  the  target  and  assessing  the  outcome  of  the  chain  of  activities.  This 
sequence  of  activities  is  at  the  heart  of  solving  every  air-to-air  engagement. 

We  have  also  discovered  that  there  is  a  need  for  a  more  detailed  description  and 
decomposition  to  fully  describe  the  Mission  Essential  Competencies,  and  to  prepare 
for  further  analyses  based  on  the  Mission  Essential  Competencies.  Specifically,  per¬ 
sonnel  that  exhibit  high  levels  of  proficiency  in  a  Mission  Essential  Competency  are 
also  proficient  in  a  series  of  sub-competencies  that  support  the  Mission  Essential 
Competency.  These  supporting  competencies  are  sets  of  high-level  skills,  many  of 
which  are  strongly  linked  to  the  pilot’s  ability  to  problem-solve.  Communication, 
adaptability  (e.g.  gameplan,  reactions,  intercepts),  weapons  engagement  zone  (e.g. 
weapons  employment),  and  decision-making  are  all  examples  of  problem  solving- 
related  supporting  competencies  and  will  be  the  focus  of  the  data  reported  here.  Some 
supporting  competencies  are  applicable  across  all  Mission  Essential  Competencies, 
and  others  are  applicable  for  only  one  or  two  Mission  Essential  Competencies.  Sup¬ 
porting  competencies  can  be  broken  down  even  further  into  knowledge  and  skills 
(e.g.  switchology).  A  variety  of  knowledge  and  skill  requirements  are  necessary  in 
attaining  a  supporting  competency  (Colegrove  &  Alliger,  2002). 
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1.1.  Syllabus  and  scenario  development 

We  have  applied  the  matrix  shown  in  Table  1  to  developing  a  comprehensive  syl¬ 
labus  for  preparing  USAF  Fighter  Weapons  School  candidates  for  the  school.  After 
a  broad  based  needs  assessment  was  completed,  we  determined  that  there  is  a  sig¬ 
nificant  need  for  principled  instruction  focus  on  specific  objectives  and  providing 
tactically  relevant  experiences  for  pilots  to  prepare  them  for  their  school  experience. 
As  researchers,  we  feel  that  DMT  offers  high-fidelity  opportunities  not  available  in 
the  current  operational  environment  especially  in  developing  problem-solving  skills, 
but  also  in  terms  of  the  sheer  number  of  opportunities  for  focused  experience  in  high 
threat,  multiple  aircraft  situations.  Moreover,  opportunities  for  dedicated  mission 
planning,  execution,  analysis  and  continuous,  focused  training  and  practice  in  a 
controlled  training  environment  don’t  exist  in  the  real  world. 

Together  with  previous  team  training  research  conducted  by  the  Army  (Alluisi, 
1991),  Navy  (Fowlkes  et  al.,  1994;  Dwyer,  Oser,  &  Fowlkes,  1995),  and  Air  Force 
(Houck,  Thomas,  &  Bell,  1991),  we  are  designing  DMT  scenarios  that  are  tied  to 
both  this  legacy  research  foundation  and  new  innovations  such  as  mission  essential 
competencies.  For  the  Fighter  Weapons  School  syllabus,  specific  learning  objectives 
have  been  identified  and  missions  have  been  designed  to  focus  on  a  number  of  sup¬ 
porting  competency  objectives  within  the  context  of  individual  and  team  capabilities 
enhancement,  of  which  the  following  directly  relate  to  problem-solving  skills  and  are 
critical  to  the  higher  level  MECs  presented  in  Table  1 : 

•  Engagement  Decision; 

•  Tactics  to  Intercept  and  Destroy  Enemy  Aircraft; 

•  Radar  Operations; 

•  Weapons  Employment; 

•  Judgment; 

•  Gameplan;  and 

•  Communication. 


1.2.  Designing  scenario  trigger  events  for  training,  research,  and  mission  success 

Once  specific  objectives  have  been  identified,  specialized  mission  scripts  and  trig¬ 
ger  events  can  be  developed.  A  trigger  event  is  defined  as  the  action  of  a  controlled 
entity  within  a  DMT  exercise  that  is  designed  to  elicit  a  specific  response  from  a 
trainee.  A  trigger  event,  properly  designed,  greatly  enhances  the  sensitivity  to  which 
certain  problem  solving  supporting  competencies  can  be  addressed  and  will  lead  to 
less  noise  in  the  measurement  system  assessing  that  competency. 

Controlled  entities  are  most  typically  computer-generated  forces  whose  actions 
are  programmed  by  instructors.  The  actions  of  computer-generated  forces  are 
therefore  programmed  to  be  tactically  valid  and  to  trigger  behaviors  from  the  trai¬ 
nee  that  fulfill  training  objectives.  For  example,  computer-generated  enemy  aircraft 
can  be  programmed  to  present  different  problems  for  each  of  the  following  the 
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objectives:  developing  and  evaluating  targeting,  tactics  and  game  plans  and  using 
appropriate  radar  operation  procedures  and  tactical  geometry.  Specific  trigger 
events  can  be  arranged  to  provide  different  and  more  complex  presentations  on 
successive  engagements.  In  one  example  scenario,  enemy  forces  may  enter  the 
engagement  as  a  single  group,  in  two  groups  separated  in  range,  as  two  groups 
separated  in  horizontal  distribution,  as  two  groups  separated  in  altitude,  or  as  a 
group  of  three  at  medium  altitude  followed  by  a  single,  high,  fast  flying  enemy  air¬ 
craft.  In  each  case,  the  trainee  must  use  their  own  aircraft’s  radar  and  effectively 
communicate  with  the  other  fighters  and  airborne  air  traffic  controllers  to  ensure 
that  all  threats  are  found,  fixed,  tracked,  targeted,  engaged  and  either  destroyed  or 
made  to  leave  the  area  of  interest. 

1.3.  Identifying  and  linking  critical  behaviors 

Given  training  objectives  and  mission  scenarios,  instructional  designers  work  with 
subject  matter  experts  to  identify  critical  behaviors  that  should  unfold  after  a  trigger 
event.  These  behaviors  demonstrate  to  the  instructor  whether  the  trainee  has  learned 
the  skills  required  to  meet  those  objectives.  Tying  scenario  events  to  training  objec¬ 
tives  and  specific  trainee  behaviors  provides  the  basis  for  instructor  evaluations  of 
team  or  individual  performance.  The  instructor  knows  for  any  given  moment  in  a 
scenario  what  objectives  are  being  trained,  what  trigger  events  are  about  to  occur, 
and  what  behaviors  are  critical  to  mission  success. 

The  ultimate  question  for  most  training  programs  is  how  the  training  affects 
actual  on-the-job  performance  (Bennett  &  Arthur,  1997).  For  military  training  pro¬ 
grams,  this  is  often  a  difficult  question  to  answer  since  most  of  the  current  military 
operations  involve  peacetime  missions.  Aircrew  members  do  not  always  have  an 
environment  where  transfer  of  training  can  be  immediately  measured.  Moreover, 
when  they  do  get  to  fly  operationally  they  do  not  routinely  have  opportunities  to 
perform  the  tasks  as  they  were  trained  in  an  environment  such  as  DMT.  For  exam¬ 
ple,  if  a  training  objective  in  DMT  is  to  provide  a  flight  leader  with  experience 
leading  a  four-aircraft  team  of  fighter  aircraft  into  combat  against  six  adversary 
aircraft,  we  can  train  and  evaluate  the  objective  in  DMT  quite  easily.  Once  the  flight 
leader  returns  to  his/her  home  unit,  however,  getting  10  aircraft  ready  for  the  same 
type  of  training  or  for  a  field  evaluation  is  both  expensive  and  impractical  in  today’s 
logistically  constrained  environment. 

However,  because  military  aircrew  training  programs  are  well-structured  and 
occur  throughout  the  career  of  an  aircrew  member,  it  is  still  possible  to  collect  data 
on  the  effectiveness  of  the  training  program  in  changing  the  behavior  that  is 
directly  relevant.  That  is,  training  evaluation  using  a  DMT  system  can  examine 
the  extent  to  which  we  have  had  an  effect  on  the  variables  that  are  ultimately  of 
most  interest  to  us.  For  DMT,  the  important  question  is  to  what  extent  does  vir¬ 
tual  training  improve  “live-fly”  performance.  Some  examples  include  the  number 
of  training  events  that  have  to  be  repeated,  instructor’s  rating  of  individual  per¬ 
formance,  and  graduation  rate  for  the  next  formal  training  program.  An  additio¬ 
nal  way  to  demonstrate  transfer  is  to  develop  in-training  transfer  tasks  that  are 
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representative  of  the  type  of  tasks  that  we  would  expect  to  see  in  combat  flying 
environments. 

DMT  training  research  is  focusing  on  extending  team  process  and  performance 
research  by  using  high  fidelity  simulations  both  as  training  instances  and  as  work 
sample  performance  tests.  Using  the  simulations  in  this  way  requires  that  very  spe¬ 
cific  information  about  the  operational  performance  environment  be  obtained. 
Therefore,  new  methods  for  capturing  critical  information  at  both  a  taskwork  and 
teamwork  level  must  be  developed. 


2.  Methods 

2.1.  Participants 

Eleven  teams  of  pilots  have  participated  in  our  USAF  Weapons  School  syllabus. 
The  USAF  Weapons  School  is  the  “graduate  school”  for  Air  Force  fighter  pilots. 
These  teams  generally  consisted  of  four  pilots  (one  for  each  of  the  four  F-16  simu¬ 
lators),  an  instructor  pilot,  and  an  air  weapons  controller.  Mean  F-16  flight  hours  in 
the  aircraft  across  all  pilot  participants  was  1269  h. 

2.2.  General  procedure 

Visiting  F-16  pilots  arrived  early  Monday  morning  for  5  days  of  training 
research.  They  were  briefed  on  procedures  and  objectives  before  beginning  a  3.5-h 
mission  routine  that  repeated  through  the  course  of  the  week.  This  mission  rou¬ 
tine  required  the  lead  pilot  to  give  a  briefing  (typically  close  to  an  hour  in  dura¬ 
tion)  to  the  flight  members  concerning  the  upcoming  mission.  During  the  next 
hour,  the  four  pilots  then  typically  flew  between  four  and  eight  air-to-air  combat 
engagements  of  the  same  mission  genre.  An  “engagement”  or  scenario  is  one  set¬ 
up  where  friendly  and  adversary  forces  were  initialized  at  greater  than  40  nautical 
miles  and  flew  until  the  either  the  learning  objectives  were  met  or  all  friendly  or 
adversary  forces  were  eliminated.  After  completing  the  final  mission’s  engage¬ 
ment,  the  pilots  had  1.5  h  in  a  debrief  facility  where  each  engagement  was 
replayed,  paused,  or  rewound  for  the  pilots  to  discuss  their  performance.  The 
debrief  facility  included  a  god’s  eye  view  and  four  multi-function  displays  from 
each  cockpit.  This  cycle  repeated  each  morning  and  afternoon  until  the  pilots 
completed  training  research  around  midday  on  Friday  (nine  total  missions). 
These  missions  followed  a  “building  block”  approach;  that  is,  missions  gradually 
became  more  complicated  as  the  missions  progressed  (e.g.  more  threats  were 
present). 

In  near  real-time,  Subject  Matter  Experts  (SMEs)  were  asked  to  evaluate  the 
team’s  performance,  utilizing  numerous  computerized  resources  at  their  disposal. 
These  included  the  god’s  eye  view,  all  pertinent  cockpit  instrumentation  for  each 
pilot,  various  instructor-operator  station  functions  (e.g.  recorded  shot  information 
and  alternative  viewpoints),  and  all  communication.  This  information  was  the  basis 
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for  the  SMEs  to  provide  subjective  evaluations  used  as  our  dependent  measures.  The 
SME  raters  used  the  following  scale: 

0:  Performance  indicates  a  lack  of  ability  or  knowledge. 

1 :  Performance  is  safe,  but  indicates  limited  proficiency. 

2:  Performance  is  essentially  correct.  Recognizes  and  corrects  errors. 

3:  Performance  is  correct,  efficient,  skillful  and  without  hesitation 
4:  Performance  reflects  an  unusually  high  degree  of  ability 


3.  Results 

Our  work  in  syllabus  development  for  the  US  Air  Force  Fighter  Weapons  School 
has  been  extremely  well  received  and  has  generated  considerable  data  on  the  benefits 
and  effectiveness  of  competency-based  training.  Participating  pilots  explicitly  reported 
that  this  DMT  learning  environment  benefited  them  by  providing  opportunities  to 
lead  multiple  aircraft  engagements,  the  opportunity  to  “see”  multi-aircraft  adver¬ 
sary  presentations,  the  opportunity  to  fight  against  advanced  adversary  missiles,  the 
benefit  of  an  advanced  digital  debrief  facility,  and  the  overall  gain  they  received  for 
the  time  spent  in  the  learning  environment. 

Results  of  a  preliminary  descriptive  analysis  of  the  SME  ratings  support  these 
assertions.  In  Table  2,  we  report  the  mean  mission  scores  given  by  the  SME  raters 
on  the  participating  pilots’  problem  solving  supporting  competencies.  The  numbers 
show  participant  group  mission  means  of  SME  ratings  for  the  entire  four-ship  of  F- 
16  pilots.  These  mission  means — unweighted  for  the  increases  in  mission  complex¬ 
ity — show  a  generally  flat  to  moderate  increase  in  those  problem  solving  supporting 
competencies.  Furthermore,  follow-up  on  the  participants  at  the  USAF  weapons 
school  by  USAF  weapons  instructors  revealed  that  there  was  a  6%  reduction  in  non- 
effective  training  sorties  and  that  the  students  showed  a  marked  improvement  in  radar 
procedures,  situational  awareness,  communications,  and  weapons  employment. 


4.  Discussion 

Environments  such  as  DMT  represent  a  major  technological  advance  in  the 
development  and  delivery  of  training  and  rehearsal  for  warfighting.  Interconnecting 
multiple,  real-time  simulators  together  with  computer-generated  forces  in  a  compe¬ 
tency-driven  and  instructionally  valid  manner  creates  an  environment  that  is 
unconstrained  compared  with  normal  training  environments  from  the  perspective  of 
the  trainee  and  at  the  same  time  highly  controlled  by  instructors.  A  well-designed 
training  exercise  will  serve  to  improve  warfighter  proficiency  on  specific  tasks  in 
order  to  meet  specific  objectives. 

The  high-fidelity  networked  simulation  environment  of  DMT  clearly  has  had  its 
benefits  for  aiding  air  combat  pilots  in  developing  higher  order  problem  solving 
supporting  competencies  defined  by  Air  Superiority  MECs.  The  self-reported  pilot 
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Table  2 

Results  across  mission  intervals  on  problem  solving-related  objectives 


Mission 


i 

2 

3 

4 

5 

6 

7 

8 

9 

Engagement  decision 

M 

2.17 

2.17 

2.58 

2.41 

2.46 

2.27 

2.70 

2.58 

2.76 

SD 

0.76 

1.18 

0.40 

0.43 

0.33 

0.49 

0.43 

0.40 

0.50 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Tactical  intercepts 

Targeting 

M 

2.58 

2.42 

2.59 

2.51 

2.64 

2.34 

2.54 

2.40 

2.57 

SD 

0.72 

0.82 

0.53 

0.50 

0.39 

0.65 

0.77 

0.58 

0.75 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Sorting 

M 

2.28 

2.17 

2.34 

2.38 

2.34 

2.35 

2.41 

2.47 

2.54 

SD 

0.75 

1.18 

0.49 

0.49 

0.44 

0.48 

0.80 

0.48 

0.62 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Intercept  geometry 

M 

2.22 

2.17 

2.25 

2.35 

2.33 

2.43 

2.53 

2.37 

2.68 

SD 

1.07 

1.18 

0.59 

0.47 

0.49 

0.59 

0.47 

0.40 

0.51 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

BVR  launch  and  leave 

M 

2.33 

1.50 

2.62 

2.32 

2.30 

2.03 

2.43 

2.50 

2.68 

SD 

0.94 

0.00 

0.21 

0.53 

0.54 

0.76 

0.58 

0.50 

0.46 

n 

2 

1 

4 

11 

11 

10 

11 

10 

10 

BVR  launch  and  react 

M 

2.44 

1.33 

2.44 

2.19 

2.36 

2.16 

2.42 

2.21 

2.69 

SD 

0.96 

0.00 

0.41 

0.59 

0.54 

0.61 

0.83 

0.67 

0.36 

n 

3 

1 

4 

11 

11 

10 

11 

9 

9 

Radar  operation  procedures 

Utilizing  correct  mode 

M 

2.67 

2.50 

2.70 

2.66 

2.49 

2.66 

2.86 

2.75 

3.08 

SD 

0.58 

0.71 

0.43 

0.59 

0.55 

0.42 

0.31 

0.39 

0.53 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Weapons  employment 

M 

2.53 

2.33 

2.37 

2.30 

2.18 

2.65 

2.60 

2.62 

2.59 

SD 

0.68 

0.94 

0.40 

0.40 

0.75 

0.36 

0.49 

0.51 

0.35 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Judgment 

M 

2.31 

2.25 

2.33 

2.47 

2.37 

2.03 

2.48 

2.29 

2.64 

SD 

0.93 

1.06 

0.45 

0.46 

0.27 

0.49 

0.69 

0.55 

0.64 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Gameplan 

Adjusting  plan  on-the-fly 

M 

2.06 

2.00 

2.07 

2.30 

2.23 

1.99 

2.14 

2.17 

2.40 

SD 

0.92 

1.41 

0.50 

0.58 

0.41 

0.55 

0.52 

0.49 

0.78 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Communication 

M 

2.17 

1.50 

2.07 

2.39 

2.40 

2.22 

2.58 

2.42 

2.61 

SD 

1.04 

0.71 

0.57 

0.66 

0.55 

0.50 

0.58 

0.65 

0.59 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

(Table  continued  on  next  page) 
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Table  2  ( continued ) 


Mission 

i 

2 

3 

4 

5 

6 

7 

8 

9 

Overall  engagement  grade  M 

2.08 

2.17 

2.10 

2.16 

2.08 

1.93 

2.05 

2.22 

2.43 

SD 

0.72 

1.18 

0.46 

0.52 

0.41 

0.61 

0.62 

0.52 

0.59 

n 

3 

2 

7 

11 

11 

10 

11 

10 

10 

Sample  sizes  for  some  cells  are  less  than  1 1  due  to  the  specific  nature  of  that  mission  or  objective.  For 
example,  missions  one  and  two  early  in  the  week  for  many  groups  frequently  required  more  fundamental 
engagements  that  prohibited  the  SME  from  rating  that  construct  for  the  entire  four-ship  of  F-16  pilots. 


participant  feedback,  instructor  pilot  feedback,  and  data  obtained  from  SME  expert 
raters  all  provide  evidence  for  this.  The  next  phase  of  our  research  will  elaborate  on 
assessing  these  problem  solving  supporting  competencies.  Extensive  effort  is  being 
undertaken  to  develop  real-time  computerized  measurement  techniques  that  capture 
and  assess  specific  knowledge,  skill,  and  supporting  competency  contexts.  These  new 
measures  will  allow  deeper  understanding  into  the  precise  knowledge  and  skill  a 
pilot  or  team  may  lack  when  they  exhibit  difficulty  problem  solving  an  engagement. 
Since  these  measures  will  be  real-time,  an  instructor  can  then  use  these  computerized 
assessments  to  optimize  scenario  selection  for  the  upcoming  mission’s  training 
objectives. 
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